New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Large-scale load testing article #15

Merged

sniku merged 7 commits into develop from large-scale-testing

May 4, 2020

Collaborator

sniku commented Apr 30, 2020

In addition to the article itself. This PR fixes the annoying codeblock indentation.

Before:

After:

The "labels" prop is no longer required for the codeblock

<div class="code-group">
This now works without crashing the project
</div>

sniku added 2 commits

May 1, 2020 00:56


          [fix] Fix the codeblock to not require label parameter

4d460ff


          first draft of the Hardware requirements article

ca1b89b

simskij suggested changes

View reviewed changes

Contributor

simskij left a comment •

edited

Loading

Stumbled upon this PR in my notification center and got curious. Thought I'd leave some comments (yeah I know it's WIP). Feel free to just resolve them if you disagree. Great and interesting article! 👏

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved


          cleanup of the hardware article

693e107

sniku changed the title ~~[WIP] Large-scale load testing article~~ Large-scale load testing article

sniku requested a review from simskij

May 4, 2020 15:02

mstoykov reviewed

View reviewed changes

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

mstoykov reviewed

View reviewed changes

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

mstoykov reviewed

View reviewed changes

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

imiric reviewed

View reviewed changes

Contributor

imiric left a comment

Nice work! The amount of performance achieved by scaling is impressive! 🎉

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

mstoykov reviewed

View reviewed changes

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated

+              ```
+              </div>
+              k6 will use about 50% of memory in comparison to running the original script. It will also reduce the CPU load.

Contributor

mstoykov May 4, 2020

I would argue that if your script is simple and uses discardResponseBodies the difference is more like 85% , with a 1.5mb json data file(If my memory served) the difference was 50%, but I can't currently find where I've said that 😭 . I would argue some links to benchmark data will be useful here.

Also another thing that compatibility-mode=base does is "significantly" increase the startup time. In the above linked script I ran the script for 60s and in the one case there were additiona minute and 5 seconds (on average) additiona time in the other a secodn and a half for 3500 VUs. Which IMO is "significant" :D

Collaborator Author

sniku May 4, 2020

Updated the text. If you have relevant benchmark links, let me know.

mstoykov reviewed

View reviewed changes

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

mstoykov reviewed

View reviewed changes

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md

+              Setup:
+              - All tests were executed on AWS EC2 instances
+              - The "discardResponseBodies" recommendation was NOT used. (results would be better with this setting).

Contributor

mstoykov May 4, 2020

I would argue you can try to do a test with this and see how much better they will be :). This way when someone doesn't use it and looks at optimizing they can see some example of what the difference si

Collaborator Author

sniku May 4, 2020

yeah, that's a fair point, but I don't want to go back to testing for this article :-)
I'll maintain this page, and will likely update it later.

mstoykov reviewed

View reviewed changes

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

robingustafsson approved these changes

View reviewed changes

Member

robingustafsson left a comment

Great article 👏 Just a few grammatical and spelling issues to fix.

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md

+              The AWS EC2 instances are relatively cheap. Even the largest instance we have used in this benchmark (m5.24xlarge) costs only $4.6 per hour. Make sure to turn off the load-gen servers once you are done with your testing. Forgotten EC2 server will cost $3312 per month.
+              Tip: it's often possible to launch "spot instances" of the same hardware for 10-20% of the cost.
+              ## Errors

Member

robingustafsson May 4, 2020

This section is really good, but it feels like it shouldn't be in its own article on "Understanding k6 errors" or something like that.

Contributor

simskij May 4, 2020

@robingustafsson I suspect there's a negation too much in that sentence? 😅 If so, I agree that it would be nice to have in its own article.

Collaborator Author

sniku May 4, 2020

👍 I'll release this as is, and then branch it out to another article :-)

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated

+              We have executed few large tests on different EC2 machines to see how much load k6 can generate.
+              Our general observation is that k6 scales proportionally to the hardware. 2x larger machine is able to generate 2x more traffic.
+              The limit to this scalability is in the number of open connections. Single Linux machine can open up to `65 535` sockets per IP.

Member

robingustafsson May 4, 2020

nitpick but it's per <src IP, src port, dst IP, dst port> tuple afaik, so should be able to go beyond 65k if machine has multiple nics with different IPs or target system hostnam is behind a load balancer with different IPs etc.

Collaborator Author

sniku May 4, 2020

I'm happy to update that, but I think I'll have to mention that k6 doesn't have support for multiple IPs and discuss this for a while. I don't want to go too much off-topic :-) (here's another hyphen!)

I also think that the limit of 65k is for <src IP, src port>. I don't think the destination IP matters when all 65k ports are already used by the load-gen server. (or maybe I'm missing something).

src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-scale k6 testing.md Outdated Show resolved Hide resolved


          Update src/data/markdown/docs/01 guides/05 Testing Guides/04 Large-sc…

9fc86d6

…ale k6 testing.md

Co-authored-by: Mihail Stoykov <[email protected]>

ppcano reviewed

View reviewed changes

src/components/templates/doc-page/code-group/doc-page-code-group.view.js Show resolved Hide resolved


          fixed plenty of spelling mistakes and addressed feedback

f51781f

Collaborator

ppcano commented May 4, 2020

1 - I am not sure if the location of the article is the best one. The Test Types section follows a structure and convention that differs from this one.

Not sure, move to MISC, Testing Guides, blog, ....?

2 - I think this article should be linked from other articles. For example fine-tuning, stress testing...

Collaborator Author

sniku commented May 4, 2020

This article is currently in "Testing Guides" not "Test types" - I think you looked in a wrong place.

2. Sure, we can link it throughout the docs once it's merged.

Let me know if you are OK with this as is or if I should change something :-)

Collaborator

ppcano commented May 4, 2020

@sniku oops. Sorry.

2 - Sure! We use the ## See also and the blockquotes patterns in other articles.

3 - I think the title could be improved to be discovered more easily from the sidebar menu.
Running large tests ?

sniku added 2 commits

May 4, 2020 21:16


          Changed title to "Running large tests"

58fe041


          more spelling mistaks corrected

31ac0ea

Collaborator Author

sniku commented May 4, 2020

Your title fits in the allotted space, so you win :-)

sniku merged commit a9fa80d into develop

simskij deleted the large-scale-testing branch

May 27, 2020 06:42

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

bolotskydev bolotskydev left review comments

imiric imiric left review comments

ppcano ppcano left review comments

mstoykov mstoykov left review comments

robingustafsson robingustafsson approved these changes

simskij Awaiting requested review from simskij

Labels

None yet